fix(ci): cosign sign container manifest by tag, not by local podman digest#539
Merged
Conversation
…igest The cosign step of the release workflow failed with MANIFEST_UNKNOWN on every release attempt after multi-arch builds were restored, e.g. on v0.45.0: Error: signing [ghcr.io/slashdevops/idp-scim-sync@sha256:2925...]: GET https://ghcr.io/v2/.../manifests/sha256:2925...: MANIFEST_UNKNOWN The previous logic resolved the digest to sign by piping `podman manifest inspect ghcr.io/...:TAG` into `jq -r '.digest // .manifests[0].digest'`. Two compounding bugs: 1. A manifest list's own JSON has no top-level `.digest` (the list's digest is computed by hashing the JSON, not stored inside it), so the `//` fallback always returned `.manifests[0].digest` -- which is the digest of the first per-arch image (arm64), not of the manifest list itself. 2. Podman re-serializes manifests on push (media-type conversion between Docker manifest.v2+json and OCI image.manifest.v1+json), so the locally computed digest does not match what GHCR stores. Cosign's lookup at that local digest therefore returned 404. Switch to `cosign sign --recursive ${IMAGE}:${TAG}`. Cosign internally HEAD-resolves the tag to the authoritative on-registry digest and signs that digest, so the resulting artifact is identical to what the broken code intended to produce. The classic "signing by tag races with concurrent pushes" caveat does not apply: this job exclusively owns the v<x.y.z> and latest tags and has just pushed them sequentially in the previous step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the
Cosign sign published container manifest (keyless / Sigstore)step of the release workflow, which fails on every release withMANIFEST_UNKNOWN: manifest unknown— most recently seen on the v0.45.0 re-attempt: Actions run 26357260675 / job 77586423614.This is the second of two CI bugs in the same job — the first (#538) restored multi-arch builds; this one fixes the signing step that was previously masked by that earlier failure.
Root cause
The step resolved the digest to sign by piping
podman manifest inspect ghcr.io/…:TAGintojq -r '.digest // .manifests[0].digest'. Two compounding bugs:.digest. The list's digest is computed by hashing the JSON, not stored inside it. So the//fallback always wins and returns.manifests[0].digest— the digest of the first per-arch image (arm64), not of the manifest list.vnd.docker.distribution.manifest.v2+jsonand OCIvnd.oci.image.manifest.v1+jsonmeans the local podman digest does not match what GHCR stores. Cosign's lookup at the local digest therefore returns 404.Net effect: cosign was asked to sign a digest that exists nowhere on the registry.
Fix
Switch to
cosign sign --recursive ${IMAGE}:${TAG}. Cosign internally HEAD-resolves the tag to the authoritative on-registry digest and signs that digest — the signature is still stored by digest, so the resulting Sigstore artifact is identical to what the broken code intended to produce.The classic "signing by tag races with concurrent pushes" caveat (which the original comment cited) does not apply: this job exclusively owns the
v<x.y.z>andlatesttags and has just pushed them sequentially in the previous step. No other actor can mutate them mid-job.Changes
.github/workflows/container-image.yml— replace the brokenpodman manifest inspect+jqdigest resolution with a singlecosign sign --recursive ${IMAGE}:${TAG}.docs/Whats-New.md— Unreleased entry.Test plan
Container Imageviaworkflow_dispatchto validate without cutting a tag.Publish Container Imagesjob::<tag>and:latest(already working post-fix(ci): restore multi-arch container builds in release workflow #538).cosign sign --recursivesucceeds for both tags.cosign verify --certificate-identity-regexp '^https://github.com/slashdevops/idp-scim-sync/.*' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/slashdevops/idp-scim-sync:<tag>succeeds locally after release.v0.45.0so the manifest gets signed.🤖 Generated with Claude Code